Depth sensor, image capture method, and image processing system using depth sensor

ABSTRACT

An image capture method performed by a depth sensor includes; emitting a first source signal having a first amplitude towards a scene, and thereafter emitting a second source signal having a second amplitude different from the first amplitude towards the scene, capturing a first image in response to the first source signal and capturing a second image in response to the second source signal, and interpolating the first and second images to generate a final image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a Continuation of U.S. application Ser. No. 14/051,605, filedOct. 11, 2013, which claims priority under 35 U.S.C. §119 to KoreanPatent Application No. 10-2012-0113259 filed on Oct. 12, 2012, thesubject matter of which is hereby incorporated by reference.

BACKGROUND

The inventive concept relates generally to sensors and methods ofoperating sensors. More particularly, the inventive concept relates todepth information calculations, depth sensors that use a time of flight(TOF) principle, image capture methods using a depth sensor, and imageprocessing systems including a depth sensor.

A “sensor” is a device that detects the state of an object andtransforms a detection result into a corresponding electrical signal.Certain sensors are also capable of communicating the correspondingelectrical signal to a variety of external circuits.

Sensors include light sensors, temperature sensors, pressure sensors,magnetic sensors, and so-called depth (or distance) sensors. The term“depth sensor” is used to denote a class of sensors that detects—as onetype of state—the location (or relative location) of the object. Sensorsmay be used to detect the state of an object in relation to a particularrange of electromagnetic signals, such as microwave signals, visiblelight signals, infrared signals, ultrasonic signals, etc. In certainapplications, a sensor will be associated with a “source” that transmits(or emits) a “source signal” or ranging signal towards an object. Theobject may then reflect a portion of the source signal, and thereflected portion of the source signal is detected by a depth sensor.

In this manner, a depth sensor may measure a depth (or range. ordistance) between the object and the sensor using a TOF measuringmethod. That is, the depth sensor may be used to measures a delay timebetween the transmission (or emission) of the source signal and returnof the reflected portion of the source signal to the sensor. In thiscontext, an area surrounding the object, an area effectively receivingtransmission of the source signal, and/or an area effectively detectedby the sensor may be termed a “scene”.

When a depth sensor is disposed relatively far from a scene, the levelof the source signal reaching the scene and/or the level of thereflected portion of the source signal returned to the depth sensor maybe relatively low. A low source signal level adversely affects thesignal-to-noise ratio for the reflected portion of the source signal.That is, the depth sensor will received a relatively large quantity ofnoise along with the relatively weak reflected portion of the sourcesignal.

In contrast, when the depth sensor is relatively close to a scene, thelevel of the reflected portion of the source signal will be relativelyhigh, thereby providing a good signal-to-noise ratio. Unfortunately,many scenes contain some objects that are far from the depth sensor andother objects that are near the depth sensor. And it is often difficultto capture images from a complex assortment of signals reflected fromboth near and far objects or scene portions. Accordingly, there is ademand for methods that increase the overall accuracy of depthinformation provided by depth sensors, particularly as such depthinformation relates to scenes having multiple objects separated near andfar from the depth sensor.

SUMMARY

According to an aspect of the inventive concept, there is provided animage capture method performed by a depth sensor, the method comprisingemitting source signals having different amplitudes to a scenesequentially; and capturing images according to source signalssequentially reflected by the scene.

The image capture method may further comprise generating a single imageby interpolating the images.

An source signal having a low amplitude from among the source signals isused to capture a point on the scene that is close to the depth sensor.An source signal having a high amplitude from among the source signalsis used to capture a point on the scene that is far from the depthsensor.

According to another aspect of the inventive concept, there is provideda depth sensor comprising a light source which emits source signalshaving different amplitudes to a scene sequentially; a light sourcedriver which drives the light source sequentially so that the sourcesignals have different amplitudes; and a depth pixel which detects pixelsignals having different pixel values according to source signalssequentially reflected by the scene.

The depth sensor may further comprise an image signal processor whichgenerates images by using the pixel signals having the different pixelvalues.

The image signal processor may generate a single image by interpolatingthe images. The light source may be an infrared diode or a laser diode.

According to another aspect of the inventive concept, there is providedan image processing system comprising the depth sensor; and a processorwhich processes pixel signals output by the depth sensor.

The image processing system may be a portable device. The imageprocessing system may be a smart TV, a handheld game console, or asecurity camera.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a depth sensor according to an embodimentof the inventive concept;

FIG. 2 is a plan view of a 1-tap depth pixel that may be included in thearray of FIG. 1;

FIG. 3 is a sectional view obtained by cutting the 1-tap depth pixel ofFIG. 2 along line I-I′;

FIG. 4 is a timing diagram illustrating signals relevant to theoperation of the depth sensor of FIG. 1;

FIG. 5 is a block diagram of a depth sensor according to anotherembodiment of the inventive concept;

FIG. 6 is a plan view of a 2-tap depth pixel that may be included in thearray of FIG. 5;

FIG. 7 is a sectional view obtained by cutting the 2-tap depth pixel ofFIG. 6 along line I-I′;

FIG. 8 is a timing diagram illustrating signals relevant to theoperation of the depth sensor of FIG. 5;

FIG. 9 is a conceptual diagram illustrating an image capture method thatmay be performed by a depth sensor according to an embodiment of theinventive concept;

FIG. 10 is a flowchart summarizing one possible image capture methodthat may be performed by a depth sensor according to an embodiment ofthe inventive concept;

FIG. 11 illustrates one possible example of a unit pixel array for athree-dimensional (3D) image sensor that may be used in certainembodiments of the inventive concept;

FIG. 12 illustrates another possible example of a unit pixel array for a3D image sensor that may be used in certain embodiments of the inventiveconcept;

FIG. 13 is a block diagram of a 3D image sensor according to anembodiment of the inventive concept;

FIG. 14 is a block diagram of an image processing system that mayinclude the 3D image sensor of FIG. 13;

FIG. 15 is a block diagram of an image processing system that mayinclude a color image sensor and a depth sensor according to anembodiment of the inventive concept;

FIG. 16 is a block diagram of an image processing system; and

FIG. 17 is a block diagram of an image processing system that mayinclude a depth sensor according to an embodiment of the inventiveconcept.

DETAILED DESCRIPTION

As noted above, figure (FIG. 1 is a block diagram of a depth sensor 10according to an embodiment of the inventive concept. FIG. 2 is a planview of a 1-tap depth pixel 23 that may be included in the array 22 ofFIG. 1. FIG. 3 is a sectional view obtained by cutting the 1-tap depthpixel 23 of FIG. 2 along line I-I′, and FIG. 4 is a timing diagramillustrating signals relevant to the operation of the depth sensor 10 ofFIG. 1.

Referring collectively to FIGS. 1, 2, 3 and 4, the depth sensor 10 iscapable of measuring a distance or a depth using a time of flight (TOF)principle. Depth Sensor 10 comprises; a light source 32, a lens module34, and a semiconductor chip 20 including the array 22. The array 22 isassumed to include a plurality of 1-tap depth pixels (detectors orsensors) 23.

The 1-tap depth pixels 23 are arranged in two-dimensional matrix to formthe array 22. Each 1-tap depth pixel includes a photo gate 110 and aplurality of transistors for signal processing.

A row decoder 24 may be used to select one of a plurality of rows inresponse to a row address provided by a timing controller (T/C) 26. Each“row” is a particular arrangement of 1-tap depth pixels in anarbitrarily defined direction (e.g., an X-direction) within the array22.

A photo gate controller (TG CON) 28 may be used to generate first,second, third, and fourth photo gate control signals (Ga, Gb, Gc, andGd) and supply same to the array 22 under the control of the timingcontroller 26.

As shown in FIG. 4, a 90° phase difference exists between the first andthird photo gate control signals (Ga and Gc), a 180° phase differenceexists between the first and second photo gate control signals (Ga andGb), and a 270° phase difference exists between the first and fourthphoto gate control signals (Ga and Gd).

A light source driver 30 may be used to generate a clock signal (MLS)capable of driving the light source 32 under the control of the timingcontroller 26.

The light source 32 emits a modulated source signal (EL) towards a scene40 in response to the clock signal. The scene 40 may generally includeone or more target object(s). The modulated source signal may havedifferent amplitudes according to driving of the light source driver 30.As conceptually illustrated in FIG. 1, various portions (and relatedobject) of the scene 40 will be separated from the depth sensor 10 bydifferent distances.

The light source 32 may be one or more of a light emitting diode (LED),an organic light emitting diode (OLED), an active-matrix organic lightemitting diode (AMOLED) and a laser diode. The clock signal applied tothe light source and/or the modulated source signal transmitted by thelight source 32 may have a sine wave or a square wave.

The light source driver 30 supplies the clock signal and/or informationderived from the clock signal to the photo gate controller 28.Accordingly, the photo gate controller 28 may be used to generate thefirst photo gate control signal Ga in phase with the clock signal, andthe second photo gate control signal Gb having a 180° phase differencewith respect to the clock signal. The photo gate controller 28 may alsobe used to generate the third photo gate control signal Gc having a 90°phase difference with respect to the clock signal, and the fourth photogate control signal Gd having a 270° phase difference with respect tothe clock signal. That is, in certain embodiments of the inventiveconcept, the photo gate controller 28 and light source driver 30 may beoperated synchronously.

The photo grate 110 may be formed of transparent polysilicon. In certainembodiments of the inventive concept, the photo gate 110 may be formedof indium tin oxide, tin-doped indium oxide (ITO), indium zinc oxide(IZO), and/or zinc oxide (ZnO). The photo gate 110 may be used totransmit near infrared wavelengths received via the lens module 34.

The modulated source signal provided by the light source 32 will bereflected in various portions by object(s) in the scene 40. It isassumed for purposes of explanation that the scene 40 of FIG. 1 includesrespective objects located at three (3) principle distances Z1, Z2, andZ3 from the depth sensor 10 (e.g., the light source 32 and/or array 22).In general, a distance “Z” between the depth sensor 10 and an object inthe scene 40 may be calculated as follows.

Where the modulated source signal is assumed to have a waveform “cosωt”, and a reflected portion of the source signal (hereafter “reflectedsignal”) (RL) received by the 1-tap depth pixel 23 is further assumed tobe “cos(ωt+θ)”, where “θ” is a phase shift or phase difference, a TOFcalculation may be made using Equation 1:

θ=2*ω*Z/C=2*(2πf)*Z/C  (Equation 1),

where C is the velocity of light.

Then, the distance Z between depth sensor 10 and an object in the scene40 may be calculated using the Equation 2:

Z=θ*C/(2*ω)=θ*C/(2*(2πf))  (Equation 2).

The reflected signal may be focused upon (or made incident to) the array22 using the lens module 34. The lens module 34 may be variouslyimplemented as a unit including one or more lens and one or more opticalfilters (e.g., an infrared pass filter).

In certain embodiments, the depth sensor 10 may include a plurality oflight sources arranged in a pattern (e.g., a circle) around the lensmodule 34. However, the descriptive embodiments presented here assumed asingle light source 32 for convenience of explanation.

The reflected signal returned to the array 22 via the lens module 34 maybe demodulated by performing an N-times sampling operation (e.g., a 4times sampling operation). In this manner, a competent samplingoperation may generate (or detect) a sampled pixel signal (e.g., pixelsamples A0, A1, A2, and A3 of FIG. 4) from the reflected signal. Thesampled pixel signals A0, A1, A2, or A3 will be described in someadditional detail hereafter.

A phase shift θ calculated by Equation 1 between the modulated sourcesignal (EL) and the reflected signal (RL) may be expressed by Equation3:

$\begin{matrix}{{\theta = {- {\arctan \left( \frac{{A\; 3} - {A\; 1}}{{A\; 2} - {A\; 0}} \right)}}},} & \left( {{Equation}\mspace{14mu} 3} \right)\end{matrix}$

where an amplitude “A” of the reflected signal (RL) may be expressed byEquation 4:

$\begin{matrix}{A = {\frac{\sqrt{\left( {{A\; 3} - {A\; 1}} \right)^{2} + \left( {{A\; 0} - {A\; 2}} \right)^{2}}}{2}.}} & \left( {{Equation}\mspace{14mu} 4} \right)\end{matrix}$

Thus, an amplitude “A” of the reflected signal (RL) may be determined bythe amplitude of the modulated source signal (EL) in the illustratedexample of FIGS. 1 and 4.

Then, and offset “B” for the reflected signal (RL) may be expressed byEquation 5:

$\begin{matrix}{B = {\frac{{A\; 0} + {A\; 1} + {A\; 2} + {A\; 3}}{4}.}} & \left( {{Equation}\mspace{14mu} 5} \right)\end{matrix}$

Referring to FIGS. 2, 3 and 4, a floating diffusion region 114 is formedwithin a P-type substrate 100. The floating diffusion region 114 isconnected to the gate of a driving transistor S/F. The drivingtransistor S/F (not shown) perform the function of a source follower.The floating diffusion region 114 may be formed by a doping with N-typeimpurities.

A silicon oxide layer is formed on the P-type substrate 100, the photogate 110 is formed on the silicon oxide layer, and a transfer transistor112 is also formed on the silicon oxide layer. The P-type substrate 100may be a P-doped epitaxial substrate.

The first photo gate control signal (Ga) is supplied to the photo gate110 during an integration interval. This is referred to as a chargecollection operation. A transfer control signal (TX) that control thetransfer of photo-charge generated within a region of the P-typesubstrate 100 below the photo gate 110 to the floating diffusion region114 is supplied to a gate of the transfer transistor 112. This isreferred to as a charge transfer operation.

According to certain embodiments of the inventive concept, a bridgingdiffusion region 116 may be further formed within a region of the P-typesubstrate 100 between regions of the P-type substrate 100 below thephoto gate 110 and the transfer transistor 112. The bridging diffusionregion 116 may be doped with N-type impurities. The photo-charge isgenerated by source signals incident into the P-type substrate 100 viathe photo gate 110.

When the transfer control signal (TX) having a “low” level (e.g., 1.0 V)is supplied to the gate of the transfer transistor 112 and the firstphoto gate control signal (Ga) having a “high” level (e.g., 3.3 V) issupplied to the photo gate 110, photo-charge generated within the P-typesubstrate 100 are concentrated in the region of the P-type substrate 100below the photo gate 110, and this concentrated photo-charge may then betransferred to the floating diffusion region 114 (e.g., when thebridging diffusion region 116 is not formed) or to the floatingdiffusion region 114 via the bridging diffusion region 116 (e.g., whenthe bridging diffusion region 116 is formed).

FIG. 3 shows a “YHA region” in which potential or photo-charge generatedwhen a high first photo gate control signal (Ga) is supplied to thefirst photo gate 110 is accumulated.

When a low transfer control signal (TX) is supplied to the gate of thetransfer transistor 112 and a low first photo gate control signal (Ga)is supplied to the photo gate 110, photo-charge is generated within theregion of the P-type substrate 100 below the photo gate 110, but thegenerated photo-charge is not transferred to the floating diffusionregion 114.

A charge collection operation and a charge transfer operation performedwhen each of the second, third, and fourth photo gate control signalsGb, Gc, and Gd is supplied to the photo gate 110 are similar to thosewhen the first photo gate control signal Ga is supplied to the photogate 110.

Although the 1-tap depth pixel 23 illustrated in FIG. 3 includes amicrolens 150 installed on the photo gate 110, the 1-tap depth pixel 23may not include the microlens 150 in some cases.

The 1-tap depth pixel 23 accumulates photo-charge during a definedperiod of time, for example, during an integration time, and outputscorresponding pixel signals A0, A1, A2, and A3 generated according to aresult of this accumulation. A pixel signal (A_(k)) generated by each ofthe 1-tap depth pixels 23 may be expressed by the Equation 6:

$\begin{matrix}{A_{k} = {\sum\limits_{n = 1}^{N}{a_{k,n}.}}} & \left( {{Equation}\mspace{14mu} 6} \right)\end{matrix}$

When the first photo gate control signal (Ga) is input to the photo gate110 of the 1-tap depth pixel 23, “k” in Equation 6 has a value of “0”.When the third photo gate control signal (Gc) is input to the photo gate110 of the 1-tap depth pixel 23, “k” in Equation 6 will have a value of“1”. When the second photo gate control signal (Gb) is input to thephoto gate 110 of the 1-tap depth pixel 23, “k” in Equation 6 will havea value of “2”, and when a phase difference of the fourth photo gatecontrol signal (Gd) with respect to the clock signal MLS is 270°, “k” inEquation 6 will have a value of “3”.

Thus, in Equation 6, the term “a_(k,n)” denotes a quantity ofphoto-charge generated by the 1-tap depth pixel 23 when an Nth gatesignal is applied with a phase difference corresponding to the variable“k”, and the natural number value of N is equal to (fm*Tint), where “fm”is the frequency of modulated source signal (EL) and “Tint” indicates anintegration time period.

Referring to FIG. 1, under the control of the timing controller 26, adigital circuit such as (e.g.,) a correlated double sampling(CDS)/analog-to-digital converting (ADC) circuit 36 performs a CDSoperation and an ADC operation with respect to the pixel signals A0, A1,A2, and A3 provided by the 1-tap depth pixel 23 in order to generate andprovide output digital pixel signals. The depth sensor 10 illustrated inFIG. 1 may further include active load circuits (not shown) fortransferring pixel signals output via a plurality of column linesimplemented in the array 22 to the CDS/ADC circuit 36. A memory 38 maybe used as a buffer that receives and stores the digital pixel signalsprovided by the CDS/ADC circuit 36.

According to the embodiment illustrated in FIGS. 1, 2 and 3, the depthsensor 10 may further include an image signal processor (ISP) 39. TheISP 39 may be used to calculate distance information or depthinformation derived from the pixel signals A0, A1, A2, and A3 stored inthe memory 38.

FIG. 5 is a block diagram of a depth sensor 10′ according to anotherembodiment of the inventive concept. FIG. 6 is a plan view of a 2-tapdepth pixel 23-1 that may be included in the array 22 of FIG. 5. FIG. 7is a sectional view obtained by cutting the 2-tap depth pixel 23-1 ofFIG. 6 along line I-I′, and FIG. 8 is a timing diagram illustratingsignals relevant to the operation of the depth sensor 10′ of FIG. 5.

Referring FIG. 5, the depth sensor 10′ may be assumed to be structurallysimilar to the depth sensor 10 of FIG. 1, except for the provision of2-tap depth pixels 23-1 in the array 22 instead of the 1-tap depthpixels 23. Thus, duplicate descriptions will be omitted.

As before, the depth sensor 23-1 of FIGS. 5, 6 and 7 is capable ofmeasuring a distance or a depth using a TOF principle.

Each of the 2-tap depth pixels 23-1 may be implemented two-dimensionallyin the array 22 includes a first photo gate 110 and a second photo gate120. Each of the 2-tap depth pixels 23-1 also includes a plurality oftransistors for signal processing.

Since the depth sensor 10′ of FIG. 7 and the depth sensor 10 of FIG. 1are different only with respect to the 2-tap depth pixels 23-1 and the1-tap depth pixels 23, the functions and operations of components andsignals indicated in FIG. 7 are the similar to analogous functions andoperations previously described, unless otherwise noted.

During a first integration interval, the first photo gate control signal(Ga) is supplied to the first photo gate 110, and the second photo gatecontrol signal (Gb) is supplied to the second photo gate 120. During asecond integration interval, the third photo gate control signal (Gc) issupplied to the first photo gate 110, and the fourth photo gate controlsignal (Gd) is supplied to the second photo gate 120.

Referring now to FIGS. 5, 6 and 7, a first floating diffusion region 114and a second floating diffusion region 124 are formed within a P-typesubstrate 100.

The first floating diffusion region 114 is connected to the gate of afirst driving transistor S/F_A (not shown), and the second floatingdiffusion region 124 is connected to the gate of a second drivingtransistor S/F_B (not shown). Each of the first and second drivingtransistors S/F_A and S/F_B may perform the function of a sourcefollower. Each of the first and second floating diffusion regions 114and 124 may be doped with N-type impurities.

A silicon oxide layer is formed on the P-type substrate 100, the firstand second photo gates 110 and 120 are formed on the silicon oxidelayer, and first and second transfer transistors 112 and 122 are alsoformed on the silicon oxide layer. An isolation region 130 forpreventing photo-charge generated within the P-type substrate 100 by thefirst photo gate 110 from interfering with photo-charge generated withinthe P-type substrate 100 by the second photo gate 120 may be formedwithin the P-type substrate 100.

The P-type substrate 100 may be a P-doped epitaxial substrate, and theisolation region 130 may be a P+-doped region. According to certainembodiments of the inventive concept, the isolation region 130 may beformed by shallow trench isolation (STI) or local oxidation of silicon(LOCOS).

During a first integration interval, the first photo gate control signal(Ga) is supplied to the first photo gate 110, and the second photo gatecontrol signal (Gb) is supplied to the second photo gate 120. A firsttransfer control signal (TX_A) controlling the transfer of photo-chargegenerated within a region of the P-type substrate 100 below the firstphoto gate 110 to the first floating diffusion region 114 is supplied tothe gate of the first transfer transistor 112. A second transfer controlsignal (TX_B) controlling the transfer of the photo-charge generatedwithin a region of the P-type substrate 100 below the second photo gate120 to the second floating diffusion region 124 is supplied to a gate ofthe second transfer transistor 122.

According to the illustrated embodiment of FIG. 7, a first bridgingdiffusion region 116 may be formed within a region of the P-typesubstrate 100 between regions of the P-type substrate 100 below thefirst photo gate 110 and the first transfer transistor 112. A secondbridging diffusion region 126 may be formed within a region of theP-type substrate 100 between regions of the P-type substrate 100 belowthe second photo gate 120 and the second transfer transistor 122. Eachof the first and second floating diffusion regions 116 and 126 may bedoped with N-type impurities.

The photo-charge are generated by source signals incident into theP-type substrate 100 via each of the first and second photo gates 110and 120.

When a low first transfer control signal (TX_A) is supplied to the gateof the first transfer transistor 112 and a high first photo gate controlsignal (Ga) is supplied to the first photo gate 110, photo-chargegenerated within the P-type substrate 100 are concentrated in the regionof the P-type substrate 100 below the first photo gate 110, and theconcentrated photo-charge are transferred to the first floatingdiffusion region 114 (e.g., when the first bridging diffusion region 116is not formed) or to the first floating diffusion region 114 via thefirst bridging diffusion region 116 (e.g., when the first bridgingdiffusion region 116 is formed).

Simultaneously, when a low second transfer control signal (TX_B) issupplied to the gate of the second transfer transistor 122 and a lowsecond photo gate control signal (Gb) is supplied to the second photogate 120, photo-charge are generated within the region of the P-typesubstrate 100 below the second photo gate 120, but the generatedphoto-charge are not transferred to the second floating diffusion region124. This operation is referred to as a charge collection operation.

In FIG. 7, the VHA region accumulates charge generated when a high firstphoto gate control signal (Ga) is supplied to the first photo gate 110,and the VLB region accumulates charge generated when a low second photogate control signal Gb is supplied to the second photo gate 120.

When a low first transfer control signal (TX_A) is supplied to the gateof the first transfer transistor 112 and a low first photo gate controlsignal (Ga) is supplied to the first photo gate 110, photo-charge isgenerated within the region of the P-type substrate 100 below the firstphoto gate 110, but the generated photo-charge are not transferred tothe first floating diffusion region 114.

Simultaneously, when a low second transfer control signal (TX_B) issupplied to the gate of the second transfer transistor 122 and a highsecond photo gate control signal (Gb) is supplied to the second photogate 120, photo-charge generated within the P-type substrate 100 areconcentrated in the region of the P-type substrate 100 below the secondphoto gate 120, and the concentrated charge are transferred to thesecond floating diffusion region 124 (e.g., when the second bridgingdiffusion region 126 is not formed) or to the second floating diffusionregion 124 via the second bridging diffusion region 126 (e.g., when thesecond bridging diffusion region 126 is formed). This operation isreferred to as a charge transfer operation.

In FIG. 7, the VHB region accumulates charge generated when a highsecond photo gate control signal (Gb) is supplied to the second photogate 120, and the VLA region accumulates charge generated when a lowfirst photo gate control signal (Ga) is supplied to the first photo gate110.

A charge collection operation and a charge transfer operation performedwhen the third photo gate control signal (Gc) is supplied to the firstphoto gate 110 are similar to those performed when the first photo gatecontrol signal (Ga) is supplied to the first photo gate 110. And acharge collection operation and a charge transfer operation performedwhen the fourth photo gate control signal (Gd) is supplied to the secondphoto gate 120 are similar to those performed when the second photo gatecontrol signal (Gb) is supplied to the second photo gate 120.

FIG. 9 is a diagram illustrating an image capture method that may beperformed by the depth sensor 10 of FIG. 1 (or analogously by the depthsensor 10′ of FIG. 5).

Referring to FIGS. 1, 4, 5, 8, and 9, the depth sensor 10 emits a firstmodulated source signal (EL′) having a first amplitude towards an objectin the scene 40 in response to a clock signal at a first point in timeT1. As a result, a first reflected signal RL′ is returned by the objectin scene 40 to the depth sensor 10. First distance (or depth)information indicating a first distance (or depth) between the depthsensor 10 and the scene 40 is generated according to a calculated firstphase shift θ between the first modulated source signal EL′ and thefirst reflected source signal RL′.

Then, the depth sensor 10 emits a second modulated source signal EL″having a second amplitude (different from the first amplitude) towardsthe object in the scene 40 in response to the clock signal at a secondpoint of time T2. Accordingly, a second reflected signal RL″ is returnedfrom the object of the scene 40 to the depth sensor 10. Here, in theillustrated example of FIG. 9, the second amplitude is assumed to begreater than the first amplitude. Hence, the amplitudes of the reflectedfirst and second source signals RL′ and RL″ correspond to the first andsecond amplitudes of the first and second modulated source signals EL′and EL″, respectively. Accordingly, the second modulated source signalEL″ and second reflected signal RL″ result in a more intense imagebrightness during subsequent display of a scene image than the firstmodulated source signal EL′ and first reflected signal RL′. In otherwords, amplitude corresponds to brightness in the working example, suchthat the greater the amplitude the more intense (or “higher”) thebrightness will be.

The first modulated source signal EL′ having the first amplitude may beused to capture a “near point” (for example, Z1) of the scene 40 that isrelatively close to the depth sensor 10. Whereas, the second modulatedsource signal EL″ having the second amplitude may be used to capture a“distal point” (for example, Z3) of the scene 40 that is relatively farfrom the depth sensor 10.

The pixel signals A0, A1, A2, and A3 having different pixel values aredetected by the depth pixel 23 by sequentially emitting the first andsecond source signals EL′ and EL″ having different amplitudes towardsthe scene 40. For example, pixel signals A0, A1, A2, and A3 each havinga first pixel value are detected by performing a sampling operation onthe first reflected signal RL′ four times, and pixel signals A0, A1, A2,and A3 each having a second pixel value are detected by performing asampling operation on the second reflected signal RL″ four times.

The first pixel value of the first pixel signal A0 detected by the firstreflected signal RL′ may be different from the second pixel value of thefirst pixel signal A0 detected by the second reflected signal RL″. Thus,the ISP 39 generates a “first image” using the pixel signals A0, A1, A2,and A3 each having the first pixel value, and generates a “second image”using the pixel signals A0, A1, A2, and A3 each having the second pixelvalue. The first image is generated in accordance with the first sourcesignal EL′ having the first amplitude, whereas the second image isgenerated in accordance with the second source signal EL″ having thesecond amplitude. A first point (or first object) in the first imagethat is relatively far from the depth sensor 10 may be distorted (ordefocused) by noise for example, whereas a second point (or secondobject) in the second image that is relatively close to the depth sensor10 may be distorted (or defocused) by noise for example. Accordingly,the ISP 39 generates both first and second images, and then interpolatesthe first and second images to generate a final image. The final imageis characterized by significantly improve quality over either one of thefirst and second images. In other words, the accuracy of distanceinformation used to generate the final image is improved.

FIG. 10 is a flowchart of an image capture method performed by a depthsensor according to the inventive concept.

Referring to FIGS. 1, 4, 5, 8, 9, and 10, the light source 32sequentially emits the first and second modulated source signals EL′ andEL″, each having a different amplitude towards the scene 40 (S10). Thelight source drive 30 may drive the light source 32 to sequentially emitthe first and second modulated source signals EL′ and EL″.

Then, the ISP 39 may be used to capture first and second imagesrespectively corresponding to the first and second reflected signals RL′and RL″ returned from the scene 40 (S20). That is, the depth pixel 23detects first and second pixel signals having different pixel values inaccordance with the first and second reflected signals RL′ and RL″. TheISP 39 captures the first and second images using the first and secondpixel signals.

Then, the ISP 39 may be used to generate a single (final) image byinterpolating the first and second images (S30).

FIG. 11 illustrates a unit pixel array 522-1 of a three-dimensional (3D)image sensor, according to an example. Referring to FIG. 11, the unitpixel array 522-1 constitutes is a part of a pixel array 522 of FIG. 13and may include a red pixel R, a green pixel G, a blue pixel B, and adepth pixel D. The depth pixel D may be the depth pixel 23 having a1-tap pixel structure as illustrated in FIG. 1 or the depth pixel 23-1having a 2-tap pixel structure as illustrated in FIG. 5. The red pixelR, the green pixel G, and the blue pixel B may be referred to as RGBcolor pixels.

The red pixel R generates a red pixel signal corresponding to thewavelengths belonging to a red region of a visible light region, thegreen pixel G generates a green pixel signal corresponding to thewavelengths belonging to a green region of the visible light region, andthe blue pixel B generates a blue pixel signal corresponding to thewavelengths belonging to a blue region of the visible light region. Thedepth pixel D generates a depth pixel signal corresponding to thewavelengths belonging to an infrared region.

FIG. 12 illustrates a unit pixel array 522-2 of the 3D image sensor,according to another example. Referring to FIG. 12, the unit pixel array522-2 constitutes a part of the pixel array 522 of FIG. 13 and mayinclude two red pixels R, two green pixels G, two blue pixels B, and twodepth pixels D.

The unit pixel arrays 522-1 and 522-2 of FIGS. 11 and 12 are illustratedby way of example recognizing that a pattern of a particular unit pixelarray and pixels that constitute the pattern may vary according toembodiment. For example, the red, green, and blue pixels R, G, and B ofFIG. 12 may be replaced by magenta pixels, cyan pixels, and yellowpixels, respectively. A color pixel array including the red, green, andblue pixels R, G, and B may be separated from an array including thedepth pixels D.

FIG. 13 is a block diagram of a three-dimensional (3D) image sensor 500according to an embodiment of the inventive concept. The 3D image sensor500 denotes a device capable of obtaining 3D image information bycombining a function of measuring depth information by using the depthpixel D included in the unit pixel array 522-1 or 522-2 of FIG. 11 or 12with a function of measuring each color information (for example, redcolor information, green color information, or blue color information)by using the red, green, and blue pixels R, G, and B included in theunit pixel array 522-1 or 522-2.

Referring to FIG. 13, the 3D image sensor 500 comprises a semiconductorchip 520, a light source 532, and a lens module 534. The semiconductorchip 520 includes the pixel array 522, a row decoder 524, a timingcontroller 526, a photo gate controller 528, a light source driver 530,a CDS/ADC circuit 536, a memory 538, and an ISP 539.

The operations and functions of the row decoder 524, the timingcontroller 526, the photo gate controller 528, the light source driver530, the CDS/ADC circuit 536, the memory 538, and the ISP 539 of FIG. 13are the same as those of the row decoder 24, the timing controller 26,the photo gate controller 28, the light source driver 30, the lightsource 32, the CDS/ADC circuit 36, the memory 38, and the ISP 39 of FIG.1, respectively, so a detailed description thereof will be omittedunless otherwise particularly stated.

According to an embodiment, the 3D image sensor 500 may further includea column decoder (not shown). The column decoder may decode columnaddresses output by the timing controller 526 to output column selectionsignals.

The row decoder 524 may generate control signals for controlling anoperation of each pixel included in the pixel array 522, for example,operations of the pixels R, G, B, and D of FIG. 11 or 12.

The pixel array 522 includes the unit pixel array 522-1 or 522-2 of FIG.11 or 12. For example, the pixel array 522 includes a plurality ofpixels. The pixels may be a mixture of at least two of a red pixel, agreen pixel, a blue pixel, a depth pixel, a magenta pixel, a cyan pixel,and a yellow pixel. The pixels may be arranged on intersections of aplurality row lines and a plurality of column lines in a matrix shape.According to an embodiment, the 3D image sensor 500 may not include theISP 539.

FIG. 14 is a block diagram of an image processing system 600 that mayincorporate the 3D image sensor 500 of FIG. 13. Referring to FIG. 14,the image processing system 600 may include the 3D image sensor 500 anda processor 210.

The processor 210 may control an operation of the 3D image sensor 500.For example, the processor 210 may store a program for controlling theoperation of the 3D image sensor 500. According to an embodiment, theprocessor 210 may access a memory (not shown) in which the program forcontrolling the operation of the 3D image sensor 500 is stored, in orderto execute the program stored in the memory.

The 3D image sensor 500 may generate 3D image information based on eachdigital pixel signal (for example, color information or depthinformation), under the control of the processor 210. The 3D imageinformation may be displayed on a display (not shown) connected to aninterface (I/F) 230. The 3D image information generated by the 3D imagesensor 500 may be stored in a memory 220 via a bus 201 under the controlof the processor 210. The memory 220 may be implemented by using anon-volatile memory.

The I/F 230 may be implemented by using an interface for receiving andoutputting 3D image information. According to an embodiment, the I/F 230may be implemented by using a wireless interface.

FIG. 15 is a block diagram of an image processing system 700 that mayincorporate the depth sensor 10 of FIG. 1 or the depth sensor 10′ ofFIG. 5, or the color image sensor 310. Referring to FIG. 15, the imageprocessing system 700 may include the depth sensor 10 or 10′, the colorimage sensor 310, which includes RGB color pixels, and a processor 210.

Although the depth sensor 10 or 10′ is physically separated from thecolor image sensor 310 in FIG. 15 for convenience of explanation, thedepth sensor 10 or 10′ and the color image sensor 310 may includephysically-overlapping signal processing circuits.

The color image sensor 310 may denote an image sensor that includes nodepth pixels and includes a pixel array including a red pixel, a greenpixel, and a blue pixel. Accordingly, the processor 210 may generate 3Dimage information based on depth information predicted (or calculated)by the depth sensor 10 or 10′ and each color information (for example,at least one of red information, green information, blue information,magenta information, cyan information, or yellow information) output bythe color image sensor 310, and may display the 3D image information ona display. The 3D image information generated by the processor 210 maybe stored in a memory 220 via a bus 301.

FIG. 16 is a block diagram of an image processing system 800 that mayincorporate the depth sensor 10 of FIG. 1 or the depth sensor 10′ ofFIG. 5. Referring to FIG. 16, the signal processing system 800 iscapable of serving only as a simple depth (or distance) measuring sensorand includes the depth sensor 10 or 10′ and a processor 210 forcontrolling an operation of the depth sensor 10 or 10′.

The processor 210 may calculate distance information or depthinformation respectively representing a distance or a depth between thesignal processing system 800 and a subject (or a target object), basedon pixel signals output by the depth sensor 10 or 10′. In this case, thedepth sensor 10 or 10′ may not include the ISP 39. The distanceinformation or the depth information measured by the processor 210 maybe stored in a memory 220 via a bus 401.

An I/F 410 may be implemented for receiving and outputting depthinformation. According to an embodiment, the I/F 410 may be implementedby using a wireless interface.

The image processing system 600, 700, or 800 of FIG. 14, 15 or 16 may beused in a 3D distance measurer, a game controller, a depth camera, aportable apparatus, or a gesture sensing apparatus. According to anembodiment, the image processing system 600, 700, or 800 of FIG. 14, 15or 16 may be implemented by using a mobile phone, a smart phone, atablet PC, a personal digital assistant (PDA), an enterprise digitalassistant (EDA), a digital still camera, a digital video camera, aportable multimedia player (PMP), a personal (or portable) navigationdevice (PND), a smart TV, a handheld game console, a security camera, oran e-book.

FIG. 17 is a block diagram of an image processing system 1200 that mayincorporate a depth sensor according to an embodiment of the inventiveconcept. Referring to FIG. 17, the image processing system 1200 may beimplemented using a data processing device capable of using orsupporting a mobile industry processor interface (MIPI®) interface, forexample, a portable device such as a PDA, a PMP, a mobile phone, a smartphone, or a tablet PC.

The image processing system 1200 includes an application processor 1210,an image sensor 1220, and a display 1230.

A camera serial interface (CSI) host 1212 implemented in the applicationprocessor 1210 may serially communicate with a CSI device 1221 of theimage sensor 1220 via a CSI. According to an embodiment, a deserializerDES may be implemented in the CSI host 1212, and a serializer SER may beimplemented in the CSI device 1221. The image sensor 1220 may be thedepth sensor 10 of FIG. 1, or the depth sensor 10′ of FIG. 5.

A display serial interface (DSI) host 1211 implemented in theapplication processor 1210 may serially communicate with a DSI device1231 of the display 1230 via a DSI. According to an embodiment, aserializer SER may be implemented in the DSI host 1211, and adeserializer DES may be implemented in the DSI device 1231.

The image processing system 1200 may further include a radio frequency(RF) chip 1240 capable of communicating with the application processor1210. A PHY (physical layer) 1213 of the application processor 1210 anda PHY 1241 of the RF chip 1240 may transmit and receive data to and fromeach other via MIPI DigRF.

The image processing system 1200 may further include a globalpositioning system (GPS) 1250, a memory 1252 such as a dynamic randomaccess memory (DRAM), a data storage device 1254 implemented by using anon-volatile memory such as a NAND flash memory, a microphone (MIC)1256, or a speaker 1258.

The image processing system 1200 may communicate with an externalapparatus by using at least one communication protocol (or acommunication standard), for example, a ultra-wideband (UWB) 1260, awireless local area network (WLAN) 1262, worldwide interoperability formicrowave access (WiMAX) 1264, or a long term evolution (LTE™) (notshown).

In a depth sensor according to an embodiment of the present inventiveconcept, and an image capture method performed by the depth sensor,multiple source signals having respectively different amplitudes may besequentially emitted towards a scene and advantageously used to increasethe accuracy of distance information in a final image of the scene.

While the inventive concept has been particularly shown and describedwith reference to exemplary embodiments thereof, it will be understoodthat various changes in form and details may be made therein withoutdeparting from the scope of the following claims.

1-13. (canceled)
 14. An image capture method performed by a depthsensor, the method comprising: emitting a first source signal having afirst amplitude towards a scene, and thereafter, emitting a secondsource signal having a second amplitude different from the firstamplitude towards the scene; capturing a first image in response to afirst reflected signal that is a reflected portion of the first sourcesignal, and thereafter, capturing a second image in response to a secondreflected signal that is a reflected portion of the second sourcesignal; and interpolating the first and second images to generate afinal image.
 15. The image capture method of claim 14, wherein a firstobject in the first image relatively far from the depth sensor isdistorted, and a second object in the second image relatively close tothe depth sensor is distorted.
 16. The image capture method of claim 15,wherein the first amplitude is lower than the second amplitude.
 17. Theimage capture method of claim 16, wherein the capturing of the firstimage in response to the first reflected signal comprises: receiving thereflected portion of the first source signal as the first reflectedsignal incident to an array of depth pixels in the depth sensor; andconverting the first reflected signal into the first image; and thecapturing of the second image in response to the second reflected signalcomprises: receiving the reflected portion of the second source signalas the second reflected signal incident to the array of depth pixels inthe depth sensor after receiving the first reflected signal incident tothe array of depth pixels; and converting the second reflected signalinto the second image after converting the first reflected signal intothe first image.
 18. The image capture method of claim 17, whereinreceiving the first reflected signal comprises focusing the firstreflected signal through a lens module, and receiving the secondreflected signal comprises focusing the second reflected signal throughthe lens module after focusing the first reflected signal through thelens module.
 19. The image capture method of claim 17, wherein each oneof the depth pixels is one of a 1-tap depth pixel or a 2-tap depthpixel.
 20. The image capture method of claim 19, wherein the depthsensor is part of a three-dimensional image sensor including at leastone of a red pixel, a green pixel, and a blue pixel, a magenta pixel, acyan pixel, and a yellow pixel.
 21. The image capture method of claim14, further comprising: storing image data corresponding to the finalimage in a memory; communicating the image data via an interface to adisplay; and generating a displayed image on the display in accordancewith the image data.
 22. A depth sensor comprising: a light source thatemits a first source signal having a first amplitude towards a scene,and thereafter, emits a second source signal having a second amplitudedifferent from the first amplitude towards the scene; and a depth pixelthat generates first pixel signals in response to a reflected portion ofthe first source signal (first reflected signal), and thereafter,generates second pixel signals in response to a reflected portion of thesecond source signal (second reflected signal); and an image signalprocessor that generates a first image from the first pixel signals,generates a second image from the second pixel signals, and interpolatesthe first and second images to generate a final image.
 23. The depthsensor of claim 22, wherein the light source is an infrared diode or alaser diode.
 24. The depth sensor of claim 22, wherein a first object inthe first image relatively far from the depth sensor is distorted, and asecond object in the second image relatively close to the depth sensoris distorted.
 25. The depth sensor of claim 22, wherein the firstamplitude is lower than the second amplitude.
 26. The depth sensor ofclaim 22, further comprising: a lens module that focuses the firstreflected signal and the second reflected signal on the depth pixel; anda correlated double sampling circuit operating with an analog-to-digitalconverter to convert the first reflected signal into the first pixelsignals and to convert the second reflected signal into the second pixelsignals.
 27. The depth sensor of claim 22, wherein the depth pixel isone of a 1-tap depth pixel or a 2-tap depth pixel.
 28. Athree-dimensional (3D) sensor, comprising: a light source that emits afirst source signal having a first amplitude towards a scene, andthereafter, emits a second source signal having a second amplitudedifferent from the first amplitude towards the scene; at least one ofcolor pixel and a depth pixel, wherein the depth pixel is configured togenerate first pixel signals in response to a reflected portion of thefirst source signal (first reflected signal), and thereafter to generatesecond pixel signals in response to a reflected portion of the secondsource signal (second reflected signal); and an image signal processorthat generates a first image from the first pixel signals, generates asecond image from the second pixel signals, and interpolates the firstand second images to generate a final image.
 29. The 3D sensor of claim28, wherein the light source is an infrared diode or a laser diode. 30.The 3D sensor of claim 28, wherein a first object in the first imagerelatively far from the depth sensor is distorted, and a second objectin the second image relatively close to the depth sensor is distorted.31. The 3D sensor of claim 28, wherein the first amplitude is lower thanthe second amplitude.
 32. The 3D sensor of claim 28, wherein the depthpixel is one of a 1-tap depth pixel or a 2-tap depth pixel.
 33. The 3Dsensor of claim 28, wherein the at least one color pixel compriseseither a first set of color pixels including a red pixel, a green pixel,and a blue pixel, or a second set of color pixel including a magentapixel, a cyan pixel, and a yellow pixel.